The First Law of Robotics - (A Call to Arms)

Even before the advent of Arti cial Intelligence, science ction writer Isaac Asimov recognized that an agent must place the protection of humans from harm at a higher priority than obeying human orders. Inspired by Asimov, we pose the following fundamental questions: (1) How should one formalize the rich, but informal, notion of \harm"? (2) How can an agent avoid performing harmful actions, and do so in a computationally tractable manner? (3) How should an agent resolve con ict between its goals and the need to avoid harm? (4) When should an agent prevent a human from harming herself? While we address some of these questions in technical detail, the primary goal of this paper is to focus attention on Asimov's concern: society will reject autonomous agents unless we have some credible means of making them safe! The Three Laws of Robotics: 1. A robot may not injure a human being, or, through inaction, allow a human being to come to harm. 2. A robot must obey orders given it by human beings except where such orders would con ict with the First Law. 3. A robot must protect its own existence as long as such protection does not con ict with the First or Second Law. Isaac Asimov (Asimov 1942): Motivation In 1940, Isaac Asimov stated the First Law of Robotics, capturing an essential insight: an intelligent agent 1 We thank Steve Hanks, Nick Kushmerick, Neal Lesh, Kevin Sullivan, and Mike Williamson for helpful discussions. This research was funded in part by the University of Washington Royalty Research Fund, by O ce of Naval Research Grants 90-J-1904 and 92-J-1946, and by National Science Foundation Grants IRI-8957302, IRI-9211045, and IRI-9357772. 1 Since the eld of robotics now concerns itself primarily with kinematics, dynamics, path planning, and low level control issues, this paper might be better titled \The First Law of Agenthood." However, we keep the reference to \Robotics" as a historical tribute to Asimov. should not slavishly obey human commands | its foremost goal should be to avoid harming humans. Consider the following scenarios: A construction robot is instructed to ll a pothole in the road. Although the robot repairs the cavity, it leaves the steam roller, chunks of tar, and an oil slick in the middle of a busy highway. A softbot (software robot) is instructed to reduce disk utilization below 90%. It succeeds, but inspection reveals that the agent deleted irreplaceable L a T E X les without backing them up to tape. While less dramatic than Asimov's stories, the scenarios illustrate his point: not all ways of satisfying a human order are equally good; in fact, sometimes it is better not to satisfy the order at all. As we begin to deploy agents in environments where they can do some real damage, the time has come to revisit Asimov's Laws. This paper explores the following fundamental questions: How should one formalize the notion of \harm"? We de ne dont-disturb and restore| two domain-independent primitives that capture aspects of Asimov's rich but informal notion of harm within the classical planning framework. How can an agent avoid performing harmful actions, and do so in a computationally tractable manner? We leverage and extend the familiar mechanisms of planning with subgoal interactions (Tate 1977; Chapman 1987; McAllester & Rosenblitt 1991; Penberthy & Weld 1992) to detect potential harm in polynomial time. In addition, we explain how the agent can avoid harm using tactics such as confrontation and evasion (executing subplans to defuse the threat of harm). How should an agent resolve con ict between its goals and the need to avoid harm? We impose a strict hierarchy where dont-disturb constraints override planners goals, but restore constraints do not. When should an agent prevent a human from harming herself? At the end of the paper, we show how our framework could be extended to partially address this question. The paper's main contribution is a \call to arms:" before we release autonomous agents into real-world environments, we need some credible and computationally tractable means of making them obey Asimov's First Law. Survey of Possible Solutions To make intelligent decisions regarding which actions are harmful, and under what circumstances, an agent might use an explicit model of harm. For example, we could provide the agent with a partial order over world states (i.e., a utility function). This framework is widely adopted and numerous researchers are attempting to render it computationally tractable (Russell & Wefald 1991; Etzioni 1991; Wellman & Doyle 1992; Haddawy & Hanks 1992; Williamson & Hanks 1994), but many problems remain to be solved. In many cases, the introduction of utility models transforms planning into an optimization problem | instead of searching for some plan that satis es the goal, the agent is seeking the best such plan. In the worst case, the agent may be forced to examine all plans to determine which one is best. In contrast, we have explored a satis cing approach | our agent will be satis ed with any plan that meets its constraints and achieves its goals. The expressive power of our constraint language is weaker than that of utility functions, but our constraints are easier to incorporate into standard planning algorithms. By using a general, temporal logic such as that of (Shoham 1988) or (Davis 1990, Ch. 5) we could specify constraints that would ensure the agent would not cause harm. Before executing an action, we could ask an agent to prove that the action is not harmful. While elegant, this approach is computationally intractable as well. Another alternative would be to use a planner such as ilp (Allen 1991) or zeno (Penberthy & Weld 1994) which supports temporally quanti ed goals. Unfortunately, at present these planners seem too ine cient for our needs. 2 Situated action researchers might suggest that nondeliberative, reactive agents could be made \safe" by carefully engineering their interactions with the environment. Two problems confound this approach: 1) the interactions need to be engineered with respect to each goal that the agent might perform, and a general purpose agent should handle many such goals, and 2) if di erent human users had di erent notions of harm, then the agent would need to be reengineered for each user. Instead, we aim to make the agent's reasoning about harm more tractable, by restricting the content and form of its theory of injury. 3 We adopt the stan2 We have also examined previous work on \plan quality" for ideas, but the bulk of that work has focused on the problem of leveraging a single action to accomplish multiple goals thereby reducing the number of actions in, and the cost of, the plan (Pollack 1992; Wilkins 1988). While this class of optimizations is critical in domains such as database query optimization, logistics planning, and others, it does not address our concerns here. 3 Loosely speaking, our approach is reminiscent of classical work on knowledge representation, which renders indard assumptions of classical planning: the agent has complete and correct information of the initial state of the world, the agent is the sole cause of change, and action execution is atomic, indivisible, and results in e ects which are deterministic and completely predictable. (The end of the paper discusses relaxing these assumptions.) On a more syntactic level, we make the additional assumption that the agent's world model is composed of ground atomic formuli. This sidesteps the rami cation problem, since domain axioms are banned. Instead, we demand that individual action descriptions explicitly enumerate changes to every predicate that is a ected. 4 Note, however, that we are not assuming the strips representation; Instead we adopt an action language (based on adl (Pednault 1989)) which includes universally quanti ed and disjunctive preconditions as well as conditional e ects (Penberthy & Weld 1992). Given the above assumptions, the next two sections de ne the primitives dont-disturb and restore, and explain how they should be treated by a generative planning algorithm. We are not claiming that the approach sketched below is the \right" way to design agents or to formalize Asimov's First Law. Rather, our formalization is meant to illustrate the kinds of technical issues to which Asimov's Law gives rise and how they might be solved. With this in mind, the paper concludes with a critique of our approach and a (long) list of open questions.